feat(alerts): hold a platform delivery until its source releases it - #123623

Problem

• An insight alert with the investigation agent on holds its notification until the agent's verdict, and a safety net sends it if the agent stalls.
• The shared alerts platform delivers every evaluation as soon as it records it, so these alerts cannot move to it yet.
• This adds a generic hold to the platform. No source sets one yet; the insight adapter does in a later layer.

Changes

• A source can put an evaluation's delivery on PlatformAlertOutcome.held_delivery (request, reason, deadline) instead of starting it.
• record_outcomes writes the hold to a new PlatformAlertDeliveryHold table in the outcome's own transaction. A recorded firing cannot lose its hold.
• release_hold in the facade releases a hold once, with send or suppress:
  • The first release wins under a row lock.
  • Only a send returns the request, and only to the winning releaser or its retry. The caller starts the delivery with start_deliveries, so the delivery keeps its usual id.
• reschedule_hold moves an unreleased hold's deadline, for a source that needs a shorter grace once it knows the outcome.
• deliver_evaluation refuses an evaluation whose hold is unreleased or suppressed, as a second guard.
• Safety net: alerts-platform-release-expired-holds runs every minute on the delivery queue.
  • It sends up to 200 holds past their deadline, within one activity payload.
  • It re-reads each deadline under the row lock, so a hold rescheduled after the sweep found it stays held.
  • It suppresses the hold of a disabled alert, so turning the alert back on never sends an old firing.
  • It still sends holds of an alert rolled back to its source, because the request records the owner at check time.
  • It starts any released send that nobody marked started 5 minutes after the release. A releaser that fails between the release and the start still delivers.
  • It prunes holds older than the 90-day history TTL.
• record_hold_delivery_started lets a releaser mark the delivery it started, so the sweep does not start it again.

• New metric alerts_platform_holds_expired_total counts safety-net sends. A rise means a source holds deliveries and does not release them.
• docs/internal/alerts-platform-temporal.md gets a "Held deliveries" section.
• Nothing user-visible changes until a source holds a delivery.

How did you test this code?

Test rationale:

• test_delivery_holds.py (new, no existing coverage for holds):
  • The hold is written with its outcome, and a replayed record adds none.
  • One release wins, and only its send returns the request, including a retry from the same releaser.
  • The sweep sends holds past their deadline and released sends nobody started. It keeps a hold rescheduled after the find, suppresses a disabled alert's hold, and prunes old ones.
  • One sweep returns no more requests than fit its payload budget.
• test_delivery_evaluation.py: one parameterized case shows an unreleased or suppressed hold sends nothing and a released send goes out. Its default configuration id became a UUID, because the guard queries a UUID column.
• Ran locally: the new tests, test_delivery_evaluation.py, test_temporal_workflows.py, test_temporal_schedule.py, and the full products/alerts_platform suite. 17 platform tests that read history from ClickHouse fail locally because the local test schema predates the source_detail column on master. CI runs them against a fresh schema.
• Not run: the sweep workflow end to end against a Temporal test server.

Release status

• No feature flag controls this change

Docs update

Updated docs/internal/alerts-platform-temporal.md.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: Claude Code, Claude Opus 5.5

• Layer I2 of the plan for moving investigating insight alerts onto the alerts platform. It is independent of the investigation-store layer (I3).
• Skills: /django-migrations, /writing-tests, /code-review high --fix, /simplify, /reviewing-with-coderabbit, /writing-pr-descriptions.
• /code-review split pruning out of the expiry lookup. /simplify moved from_stored onto Temporal's own converter, tied hold retention to the history TTL, and kept a failed prune from blocking the sweep.
• PostHog Review findings, all fixed: the safety net now re-reads the deadline under the lock, recovers a send whose start was lost, and bounds its result by payload size.
• Suppressing a disabled alert's hold, instead of skipping it, is the owner's decision.
• CodeRabbit CLI, --deep: one finding, fixed. A hold reason longer than its column, or a negative deadline, is now rejected when the hold is built instead of failing the batch record.
• No duplicate: an open-PR search for delivery holds found nothing.